Of Stack

centos6.5 Installation es1EN1.2.1 Tutorial Details [Pro test version]

  • 2020-10-23 21:18:44
  • OfStack

This section covers the installation steps only briefly

1. Role assignment 


10.11.84.4 web-crawler--1.novalocal master/slave
10.11.84.5 web-crawler--2.novalocal slave
10.11.84.6 web-crawler--3.novalocal slave

2. Installation directory /usr/local/hadoop (HADOOP_HOME)

Create group hadoop

groupadd hadoop

4. Create user hadoop 


useradd -d /home/hadoop -g hadoop hadoop
chown -R hadoop:hadoop /usr/local/hadoop

5. Install jdk per machine (install in the same directory)

1) Check the VERSION of java, it can be seen that jdk is openjdk, and it is better to change it to sun jdk 


[hadoop@hadoop2 ~]$ java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-b12)
OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)
[hadoop@hadoop2 ~]$ rpm -qa | grep java
java-1.8.0-openjdk-headless-1.8.0.131-11.b12.el7.x86_64
javapackages-tools-3.4.1-11.el7.noarch
tzdata-java-2017b-1.el7.noarch
java-1.7.0-openjdk-headless-1.7.0.141-2.6.10.5.el7.x86_64
java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64
java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64
python-javapackages-3.4.1-11.el7.noarch

2) Uninstall openjdk

rpm -qa | grep java | xargs yum -y remove

3) Install sunjdk

Extract jdk-8u181-ES50en-i586. tar. gz and name jdk under usr/local directory

Configure the environment variables as follows

Add the following at the end of vim /etc/profile 


export JAVA_HOME=/usr/local/jdk
export JRE_HOME=${JAVA_HOME}/jre
export PATH=${JAVA_HOME}/bin:${PATH}
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib:${CLASSPATH}

4) Check whether jdk has been installed successfully 


[root@hadoop2 local]# java -version
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

6. master to slave ssh password-free login

Execute the following commands under each machine hadoop user 


[hadoop@hadoop1 ~]$ ssh-keygen -t rsa -P '' #1 Enter directly 
 send master  The machine ~/.ssh/id_rsa.pub  To all slave node 
Scp id_rsa.pub hadoop@10.11.84.5:~
Scp id_rsa.pub hadoop@10.11.84.6:~
 Let after each set slave node hadoop  The user executes the following command 
Cd ~/.ssh
Cat ../id_rsa.pub >> authorized_keys
Chmod 644 authorized_keys

7. Close all machine firewalls, selinux

1) Less than centos 7

A firewall
View firewall status
service iptables status
Temporary shutdown of firewall
service iptables stop
Permanently close the firewall
chkconfig iptables off
selinux
setenforce 0 # temporarily closed
Permanent closure requires modification of the following file
vim /etc/selinux/config
Change SELINUX=enforcing to SELINUX=disabled

2)centos 7

firewall-cmd --reload # Restart firewall
systemctl stop.service # stop firewall
systemctl disable.service # forbid firewall to boot
firewall-cmd --state # View default firewall status (notrunning on off, running on off)
selinux ditto

8. Modify the hadoop profile 


Core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl" rel="external nofollow" rel="external nofollow" rel="external nofollow" ?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/tmp</value>
<description>Abase for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://web-crawler--1.novalocal:9000</value>
</property>
</configuration>
hadoop-env.sh  add java_home
 export JAVA_HOME=/usr/local/jdk
hdfs-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl" rel="external nofollow" rel="external nofollow" rel="external nofollow" ?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/home/hadoop/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/hadoop/data</value>
</property>
</configuration>
mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl" rel="external nofollow" rel="external nofollow" rel="external nofollow" ?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
 <property>
    <name>mapred.job.tracker</name>
    <value>http://web-crawler--1.novalocal:9001</value>;;;
  </property>
<property>
    <name>mapred.system.dir</name>
    <value>/home/hadoop/mapred/system</value>
  </property>
<property>
    <name>mapred.local.dir</name>
    <value>/home/hadoop/mapred/local</value>
  </property>
</configuration>

5)masters

web-crawler--1.novalocal

6)slaves 


web-crawler--1.novalocal
web-crawler--2.novalocal 
web-crawler--3.novalocal

9. Modify the environment variables to add HADOOP_HOME and JAVA_HOME 


Vim /etc/profile (root)
export ANT_HOME=/usr/local/ant
export PATH=${PATH}:${ANT_HOME}/bin
export HADOOP_HOME=/usr/local/hadoop-1.2.1
export JAVA_HOME=/usr/local/jdk
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
export CLASSPATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
export PATH=/usr/local/lib/phantomjs/bin:$PATH
export ANT_HOME=/usr/local/ant
export PATH=${PATH}:${ANT_HOME}/bin
export PATH=$PATH:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin

10. Start hadoop

1) On master format namenode

hadoop namenode -format

2) start hadoop

sh /user/local/hadoop/bin/start-all.sh 

11. Run a simple test 


Hadoop fs -mkdir -p /user/hadoop/private/in
Hadoop fs -put /user/local/hadoop/bin/start-all.sh /user/hadoop/private/in
Cd /usr/local/hadoop
Hadoop jar hadoop-examples-1.2.1.jar wordcount /user/hadoop/private/in /user/hadoop/private/out

conclusion

Related articles: